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1 . THE MAJIYLAND ACCOUNTABILITY PROGRAM 



Accountability means many things to many pe^^ple. To some it means 
getang rid ^31 incompetent teachers; to others it means testing; still 
others view it: as a system for improving the quality of education. As of 
1974, thirty states had enacted legislation which contained approaches to 
accountability. (1) Many of these laws deal with the assessment of programj 
aad personnel with an ultimate goal of improving the performance of learner? 

Maryland is or\e of the states that enacted a rather comprehensive 
accountability law in 1972. This was done in response to a growing public 
concern about tie quality of education and a need for better Information 
about what students learn in schools. 
The purposaa of this Act are: 

...♦.to assure that educational programs operated in the 
public schools; of Maryland lead to the attainment of 
estiblished objectives for education, to provide information 
for accurate analysis of the costs ass^c/ated with public 
edu:ation programs and to provide information for an 
analysis of the differential effectiveness of instructional 
pro »rams. (2) 

The State :)epartment of Education, charged with implementing this 
law, developed a four stage plan for this purpose. This includ^: 

1. The de-'elopment of statewide- goals for reading , writing, and 
mathem.itlcs . 

2. Tlio pri'paratlon of local goals and objectives in conformity 
with the state goals by each of the 24 county school systems. 

3. The idintificatlon and administration of instruments (Iowa 
Tests c.f Basic Skills and Maryland Functional Reading Test) 
to assi ss achievement of these goals in four grades. 

A. The preparation of an annual report on the implementation of 

the pre gram, the results of assessment and further recommendations. 
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XI.' INCEPTION AND PURPOSES OJ^ THE 
AtTEKNATIVE ACCOUNTABILITY PROGBAM 



Almost from itn inception this program was criticized, particularly 
by teachers. The basis for their objections was fourfold. 

1. The goals and objectives were from the top down (state to local) 
which were not always suitable for local needs. \ 

2. The burden of accountability (blaine for poor achievement) was 
iiq)llciply placed on teachers rather than being shared by others 
that lD|$lti«nced education. 

3. The aaQessment instruinent ^id not accurately measure what was 
being taught in all schools. 

A. In api^e of a regression analysis to adjust scores for certain 

inherex?t variables, test resultai are not necessarily comparable / 
amot^g schools. . 

Discussions between representatives of the Maryland State Department 
of Education (HSDE), the Maryland State Teachers Association (MSTA) and the 
Nalipnal Education. Association (NEA) resulted in the planning and Joint 
funding of the Jflarylaad Alternative Accountability Pilot Project (IIAAPP) . 
The philosophy underlying this project was that the individual school 
(sta^f, students, parents and community) should determine what they want to 
accomplish, hiow they will assess these accomplishments tod how they will 
report the results to interested parties. 

The major purposes of the MAAPP were: 

1. To increase comnunity involvement, particu rly in goal setting, 
and develop more meaningful communication with various aspectc 
of the community.' 

2. To develop and/or select alternative approaches to assessment. 

3. To utilize assessment results to improve instruction. 
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IIT. SCHOOLS, FUNDING AND MANAGEMENT 

All local school systems were infonned of the project and invited 
to participate. Seven schools from four different systeias chose to 
pnrt:<.cipate (five elementary, one high school and one special education 
center). One elementary school later withdrew. Each school. received 
$1500 to use in purchasing materials^ hiring substitutes to release 
teachers, paying teachers-for extra services or to otherwise use in 
conducting the project. In addition, each one received ei^t days of 
services from an assessment conaultaatrT^These monies, plus others for 
the overall operation of the project, were provided jointly by the three 
sponsoring organizations (MSDE, MSI \ and NEA) . 

Initially, representatives of these three agencies cooperatively 
managed the project. They provided general guidelines for schools to 
follow and jointly met witVi the principal and staffs of the participating., 
schools to assist them in their task. A graduate student from the state 
university was employed as a part time field coordinator to keep records, 
foster communications aad assist in expediting work. However, changes in 
personnel and schedule conflicts made this form of management unworkable. 
Therefore, it was decided to make the MSDE representative the project 
manager to coordinate activities and administer the project. Tlie manaj^er, 
working with the assessment consultants and the field Coordinator proved 
to be a successful way of operating the project. 



IV. CONCEPTUAL MODEL 

rne KAA?? was conceptuailzed as an eight-^stage model. This model is 
depicted in TJgure !• (3) Stage one is the deSrelopmnt and refinement .of 
objectives In the three basis areas o^f accountability (reading, writing and 
mathematics). In essence, this was a currictdum dev6lo|>taent task. These 
objectives formed the scope and sequence for these three areas and provided a 
basis for selecting the content to be taught as well as the criteria for 
evaluation. 

The second stage was broken down into two activities: (A) the identi- 
fication of possible ways to assess each activity and (B) the organization 
or classification of each objective according to these various ways of 
assessment. 

Stage three called for the utilization of the previous stage in 
actually selecting or developing the assessment Instruments, 

Stage four required schools to plan who to assess and when. 

A decision had to be made whether to assess every student : every 
objective, to saiiqple either students or objectives, or both. The step also 
necessitated a decisiion on whether to evaluate all students in a relatively 
short period of time oi: to spread the assessment process over the entire' year. 

The final stage (eij^t) is the primary reason for the existence of an 
accduntability program; the use of these results to improve instruction. 

The balance of thia paper will be devoted to describing each of these 
stages as they we developed and implemented in the six pilot schools. 
The successful lirocedures developed by various schools will be highlighted 
as well as the problems that aroise. Recommendations will be made abased 
on this experience for others who may wish to use a slmillar plan. 
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Figure 1 



MARYLAND ALTERNATIVE ACCOUNTABILITY PILOT PROJECT ASSESSMENT MODEL 



STAGE 8 




Use of data 
to improve 
instruction 



STAGRJ !A 



Idc^ntify an 
assessment 
procedure for 
each objective 
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V. DEVELOPMENT OF OBJECTIVES (stage One) 

All schools in the state were required to write objectives for 
reading, writing and tiAthematica that were consistant with the goals of 
their county and the state. In the MAAPP schools careful consideration 
had to be given to "the measurability" of these objectives since the 
schools themselves would be responsible for their assessment. Schools in 
this project were also strongly encouraged to involve parents and members 
of the community In the process. (3) All schools involved parents and/or nembfcrs 
of the community in the early part of this task. ^Some' schools organized small 
on-going groups of parents and teachers to set goals. Others mailed invltat/ons 
to selected participants. One held a mass meeting cf parents to get involve- 
ment; TWO schools usr^d a mall questionnaire to get parents' responses. While 
community involvement was generally good at the onset, it tended to atrophy 
as the work got down to the "nitty gritty" of specific objectiye writing. 
At this point, teachers and consultants took over the job. 

Some schools organized by grouping teachers according to grades, 
levels or departments (depending on individual school's organization). 
Willie rthers organized according to subject areas across grades. Whichever 
pattern was used for the primary writing task, meetings were held using 
the opposite organization to provide for communication and articulation. 
For example, if teachers at a grade level met to write objectives in one 
or more areas, representatives of each grade met to articulate sequence 

■y \ « 

bstween grades. 

Small task groups proved to be most productive. Meeting times varied 
among the schools. Some net during the day using substitutes to xrover classes, 
some vuet at the end of the school day and some meetings were held on Saturday. 
The latter seemed to be most productive as teachers were fresh and free " 
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from concern about their clashes. 

This task was not an easy one for the schools. Alfhou^ it was suppose 

to be completed during the first half of 1975, When the ac esssaiut 

consultants were employed in August they found considerable work needed 

to be done to make the objectives "assessable". This included not only 

\_ . ■ ■ • 

cleaning up the stsceTaent of the objectives, but also in helping teachers 

clarify what they wanted to teach and ironing out duplication and gaps 

between grades. The input of the consultants was most valuable in this 

process. The task was not completed until the end of 1975 for most schools 

and even ran into, the spring of 1976 for some. 

Although the task was difficult, some valuable outcomes accrued.. Firs 

\ ■ f 

teachers learned aeveral things: to write objectives in measurable terms; 

more about their subject matter, particularly its organization; and they 
improved their corannmication with one another, perents and the administra- 
tion. Second, a more complete and better organized curriculum evolved 
from this process. 

A comparison of the objectives produced at the elementary schools 
indicates a hi^ degree of similarity. The question must: be raised if 
thia process of "rc*.discovering the wheel" is worth the effort. One 
recommendation that could resolve this issue is that a bank of objectives 
be collected at the state level and made available to local schools. They 
would have the privilege of selecting objectives from this bank, creating 
their own or doing some of each. 
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VI. IDENTIFICAT^j, ORGANIZATION AND DEVELOPMENT 
OF ASSESSMENT PROCEDURES, (stage two and three) 



Stage two A (Identification of possible ways to assess the objectives) 
and two B (organizing objectives according to assessment procedures) were 
neither dlfflcxilt tasks to accooxpllsh nor were they extremely useful. 
While a list of possible ways of a^asesslng.each objective was quickly 
developed and objectives classified according to this list, not until 
the teachers got Into the tasks of selecting or developing specific 
instriiments (stage three) did the whole thing begin to fit together. 
Only in the special edjucation center, where the decision to rely heavily 
on check lists was made in advance, did stage two prove to be very useful. 
As teachers explored existing instruments and worked on development of 
their own, preliminary deqjJLsions were either changed or reinforced. 

The task of selecting and developing instruments for assessment 
was as time consuming, although probably not as frustrating, as developing^ 
the objectives. In many schools teachers reviewed standardized tests, 
criterion-referenced tests associated with textbook series and criterionr • 
referenced items obtained from the Instructional Objectives Exchange. 
Some items were selected from these sources. In addition, teachers 
developed criterion- referenced items and observational checklists for 
certain objectives. (4) 

Other than the heavy time commitment, no other problems seemed to 
be evident in this process. Many teachers reported that their work on 
this task improved their writing skills and increased their understanding 
of the relationship between objectives and the evaluation of student achiev.»t* 
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VII. WHO, WHAT AND WHEN (stage Four) 

A major conceTn in this area was whether or not to aasefts every 
student on every objective or to sanple some students and/or objectives. 
Despite the work Involved, teachers favored assessing every child. Their 
goal was to obtain diagnostic data on students which could be used for 
planning future instruction as well as for reporting to parents and 
.others. 

While teacheri also favored assessing students on all objectives, 
the time pressure i^rented by the number of start up tasks in the initial 
year made this impossible in most schools. Therefore, most of them 
sampled theynuuber of objectives upon which they would asisess students 
with anticipation of expanding the nuvdber they w^uld assess the next year. 



that had to be resolved, 
throughout the year v?a8 



The time of assessment was another, issue 
Most teachers agreed that on-going assessment 

desirable. This wou3-d enable them to use thej results as feedback to 
themselves, students, parents ^and other parties so that changes could be 
made that would facilitate leamiag. However, this was not possible during 
the initial year when the preparation for assessment consumed the first 
six months. Some schools chose to assjess children lover a few days period; 
others made it an on-going process lasting up to two months. Some set 
aside a portion of the school day for assessment; others observed the child* 
behavior and work products throughout the day. 

While most of the effort went into student assessment, the larger 
concept of accountability as responsibility forced some consideration to 
be given to who waa responsible for the success or failure of students. 
Teachers were obviously reluctant to accept total responsibility for 
student achieivement when so many other factors such a6 family, community 




support, school resources and sdninlotratlon impinged on this ^utcbne. 

Unfortunately, time constraints and priorities established by the model 

left little opportunity to consider this idea. Only one school did any- 

thing about it. In this school, two instruments were developed - one for 

evaluation of the principal and one for teacher evaluation. Teachers 

\anonymou8ly completed the principal evaluation form and teachers utilized 

\ ' 1 
/ fhe other form to perform a self-evaluation. Although these steps v^re 

; • I • 

small, they at least initiated action on a "total accountability program". 
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VIII. ASSESSMENT (stage Five) — 

At this stage the Instnsnente and procedures developed in the previous 
two stages were implemented. In roost schools objectives were evaluated on 
some form of a "mastered - not mastered" basis. Some schools used a 
modification of this by including a "partially learned" category. One — 
school used only a percentage correct systemw 

Criteria werr. established, if they were not Inherent in the pblectives, 
to deteiTnine what was meant by mastery.. Record forms were devised and 
utilized to collect the data on each cillld. In most ca^es, these took 
the form of class summary sheets on which each child's performance on 
each objective was recorded. 

IX, ANALYSIS OF RESULTS (stage six) 

Results were analyzed both individu !ally and by groups. Some schools 
kept records ^only on each student indicating the number of objectives mastert a 
or the score each obtained on given tests. Others grouped these data and 
determined the percentages of each grade that mastered each objective 
or the averagi^ score and variability for each group on each test. 

Some schools hand-scored the assessments at the classroom level 
while other used JBM answer sheets with computer scoring. 



/ 
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X. SEIPORTING THE RESULTS (stage seven) 

By June ] 76 the majority of MAAPP schools had reached Stage seven, 
reporting the school level and individual student a88eB8n.ent results. 
The sponsoring agencies presented MAAPP results to the appropriate 
lekislatlve bodies and' ta the coamnmlty at large while the schools 
concentrated on ways to explain their programs and results to parents. 
Some schools held volunJ ary parent conferences with teachers to discuss 
the students' achieves ,t. As an alternative to the ti«e-consu»lng task 
of individual conferences, several schools planned evening meetings with 
parents. During these sessions, the principal and teachers reported on 
assessment results and explained how the information would improve educational 
planning_and inatructibn. Another approach to reporting resuUs was a 
take-home report card, also calle<i an "objective mastery card", which 
indicates each child's level of achievement of objectives assessed. One 
, school Intends eventually to dispense with its regular report card and replace 
It with the "objective mastery card." These assessment results remain in each 
students' folder as the student progresses through the skills required at each 
grade level. ' 

Presenting the. results in an understandable fashion is a key to the 
succes.s of this program. Parents, students and the public at large have for 
so long been^ conditioned to broad subject letter grades and comparison teat 
•scores (norm-referenced) that the new mastery of objective type of evaluation 
(criterion-referenced) requires considerable explanation.'' 
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XI. USE OF DATA 10 IMPBOVE INSTWICTION (Stag^ Eight) 

This final stage of the MAAPP was only touched upon by some schools 
because tliae ran out for the school year. A fev teachers-ia schools 
where testing vas continuous over a period of two ntonths indicated they 
used some of the eiarly results to reteach sone objectives. Other 
schools indicated that some objectives In which students did not do so well 
were marked for special attention the following year. Some schools 
remarked that they not only planned to look at Instruction, but also 
at the objectives themselves, the assessment instruments and the record 
system. 

Two schools planned summer workshops with the anticipation of com- 
pie ting the assessment development tasks and reviewing the results of the 
past year. 

It was probably too much to expect a school staff to complete the 
entire eight steps In one year. However, a sufficient amoimt was accomplished 
to show the viability of the model. 
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XII, OTHER ASPECTS OF PROJECT 

Tliere were several other aspects of the project that contributed 
to Its success. One was meetings of the assessment consultants, field 
coordinator and project manager. At these meetings ideas were exchanged, 
time tables set, problems discussed and a plan for the evaluation of the* 
project developed. No attempt was made to develop uniformity among the 
schools; in fact diversity was encouraged in order to see what worked. 
Hcjwever, the shared expertise of this group was helpful in resolving 
many issues. 

Another key factor was the role of the field coordinator. This person 
spent considerable time communicating with the schools, the consultants 
and the manager, expediting many details. 

The project manager provided overall leadership to the project. The 
enthusiasm and p^^rseverabce of this individual was central in keeping 
people moving when other priorities attempted to get in the way. 

A final factor that was helpful in moving the project was a series 
of intra-school meetings. Principals - ^y teachers of the participating 
schools met to share ideas, discuss problems and mutually reinforce each 
Other. \ y 

Two disappointing aspects of the project were the lack of input and 
concern of representatives of the sponsoring agencies (other than the one 
which the project manager represented), and tlje county l^vel personnel. 
After the assessment consultants were brought on board, the sponsoring 
groups seemed to feel they were relieved of all responsibility except 
funding and publicising the fact that it was "their" project. Other than 

16 
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level of adodnlstratloa. Ifore "drum beatlag «nd horn tooting** by the 
achool and staft aay have altexed thla» but they were too busy with the 
t^'ok at hand. ^ ' 

XIII MAJOR PROBLEMS 

There were several major problems that arose during .the project. 
These included. tiliaV flnaacial assistance » tecdmical aesistance, consistent 
leadership and folloir up. - 

The process of developing objectives » which was tantamount to a major 
curriculum revision » was extremely time consuming. Part of this was due 
to the lack of expertise and leadership at the school level during the 
initial phase of the project. This was remedied in part. when the 
assessment consultants started working* Howevier» a good deal of it had 
to do with the scope of the task as envisioned by the authors of the pro- 
ject. It was assumed that the curriculum in each school was in order and 
the obji^ctives of this curriculum needed only to be translated into 
behavioral terms. Unfortunately » the curriculum in most American schools 
lis not In good order. The scope and sequence is not well laid out and 
many teachers are *'dolng their own thing**. Many curricular problems had 
to be resolved in these schools before the objective task could really be 
• done well. Some problems were resolved by communication among staffs some 
by consensus » some by coiq>roml8e and so^ by majority vote. Any group 
contjemplating a project similar to the MAAPP should be aware that a sound 
curriculum precedes assessment. The conltinsnt needed to do this requires 
more than a few hours of released tlms or an afternoon once a ^ '.ek. 

Time was also 'a problem in accomplishing the other stages. The fact 
that most schools only assessed a sample o£ their objectives Indicates 
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the crunch they were in. Hindsight would suggest that a full calendar 
year> Including a aunmer, should be devoted to curriculum and assessment 
development before inplementlng the project even on a pilot basis. 

Financial assistance was a second problem. The $39,000 inve.sted in 
the project produced a tremendous amount of output. Additional funds 
were needed to buy more tixse, particularly for staff and consultants. 
Some of the shortcomings of the project may be tra-ed to this. One school 
%d.thdrew from the project because they felt tliey could iiot "complete a 
creditable project with the time and resources available". 
\ It became obvioua early in the project that teachers lacked the 
technical expe: Ise to do a good job in curriculum and assessment develop- 
ment. This should not be construed as a criticism of teachers. Their job 
is to teach, rot wilte curriculum and constrpct tests. The employment of 
consultants who worked with the staff in doing the job resolved this issue. 
Consultants for such a task must have not only the technical knowledge, but 
be willing to "roll up their sleeves" and get down to work with th** staff. 
Mudi of their task consisted of providing examples, locating resources, 
technical editing and revision and actually doing some of the writing • 

Changes in leadership proved to be another major problem. The 
resignation and replacement of the field coordinator and two shifts in the 
administrator of the Maryland Accountability Program, who represented MSDE, 
interrupted the project. Schools became confused and questioned the commltmeni 
of the agencies. However, the consummation of the position of project manager, 
the retention of a new field coordinator and th^ employment of conoultants t 
all of vAiom stayed with the project for the entire pilot year provided the 
consistency nefeded to enable the schools to accomplish the great amount of 
work they did. 
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At the cqapletian of the pilot year, it was agreed that the schools 
vould continue refining objectives, developing the rest of the assessmsnt 

\ * 0 

tasks and fusing the results to Itqprove Instruction. One school received a 
county grax^t to do thiji in a sumer vorkshop. IWo factors Interf erred 
with this plen, however. Although the sp^nso^rlng agencies agreed to. fund 
the schools for some expenses and consultant help during t;he follow-up 
year, it was not forthcoming. The second factor was the resignation of 
the project aanager to take another position. His replacewent did not 

have the saoK enthusiasai for the project. This lack of follow up is 

■J 

having a distinct affect upon the schools conmltii^t to continue work on 
the task. 

XIV EVALUATION 

The purpose of t'ne MAAPP was to demonstrate how an accountability 
prograjn could work from the grtts roots level. An evaluation was designed 

and conducted by the manager and consultants to deternine how well it worked. 

^ * , ■ ■ " • ■ ■ 

This is the subject of a second paper to follow. This paper has described 

r 

the process, the resources and the products, illustrating some of the succesfc 
and sooe of the problew encountered. 
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